Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Michael Stapelberg]

Okay.

Given the severely constrained manpower from all involved sides, I think
the best path forward to make some progress is to continue the review (and
eventual merge, hopefully) of my proposed patch in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859867#105. If we want to
expand scope, we can do that later.

josch, could you take another look at the patch, please? Thank you.

On Fri, Jan 5, 2018 at 2:09 PM, Raphael Hertzog  wrote:

> On Fri, 05 Jan 2018, Michael Stapelberg wrote:
> > Raphaël, Benjamin, what’s the current status? It’s been quite a number
> > of months since https://bugs.debian.org/859867#122, and I’m still very
> > interested in having sbuild easily installable.
>
> Benjamin never answered on the willingness to host this new infrastructure
> within packaging-dev.
>
> As for me, while I'm still supportive of the idea, I'm afraid that I won't
> be able to contribute code in the near future. I have other priorities
> related to tracker.debian.org. Hopefully my ideas will help you design
> something modular enough to cover the needs of derivatives. Please keep
> me in the loop, I might be able to help once it becomes a bit more
> concrete and then I might be able to invest some time as part of my work
> on Kali.
>
> Cheers,
> --
> Raphaël Hertzog ◈ Debian Developer
>
> Support Debian LTS: https://www.freexian.com/services/debian-lts.html
> Learn to master Debian: https://debian-handbook.info/get/
>



-- 
Best regards,
Michael

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Raphael Hertzog]

On Fri, 05 Jan 2018, Michael Stapelberg wrote:
> Raphaël, Benjamin, what’s the current status? It’s been quite a number
> of months since https://bugs.debian.org/859867#122, and I’m still very
> interested in having sbuild easily installable.

Benjamin never answered on the willingness to host this new infrastructure
within packaging-dev.

As for me, while I'm still supportive of the idea, I'm afraid that I won't
be able to contribute code in the near future. I have other priorities
related to tracker.debian.org. Hopefully my ideas will help you design
something modular enough to cover the needs of derivatives. Please keep
me in the loop, I might be able to help once it becomes a bit more
concrete and then I might be able to invest some time as part of my work
on Kali.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Michael Stapelberg]

Hi,

Johannes Schauer  writes:

> Quoting Simon McVittie (2017-08-20 01:16:12)
>> On Sun, 20 Aug 2017 at 00:26:26 +0900, Johannes Schauer wrote:
>> > since sbuild already supports autopkgtest as a backend (and thus also
>> > autopkgtest-virt-qemu which vectis seems to be using) I'd be especially
>> > interested in hearing which features are missing from sbuild that prevent
>> > vectis from making use of that functionality.
>> 
>> The reason vectis doesn't do this is a design choice, not an omission
>> in sbuild.
>
> Thanks for your explanation! I now see how vectis is indeed doing something
> that is not in the scope of sbuild itself and that vectis indeed has to be a
> wrapper around sbuild and other tools.
>
> Though maybe it would be a good idea to add parts of your explanations from
> your mail to the README.md and package description. It seems at least me and
> Raphael were fooled into thinking that vectis' purpose was a different one 
> than
> it actually is.
>
> Thanks for writing this!

So, I take it vectis is out of the picture for the purpose of this bug.

Raphaël, Benjamin, what’s the current status? It’s been quite a number
of months since https://bugs.debian.org/859867#122, and I’m still very
interested in having sbuild easily installable.

Thanks!

-- 
Best regards,
Michael

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Johannes Schauer]

Quoting Simon McVittie (2017-08-20 01:16:12)
> On Sun, 20 Aug 2017 at 00:26:26 +0900, Johannes Schauer wrote:
> > since sbuild already supports autopkgtest as a backend (and thus also
> > autopkgtest-virt-qemu which vectis seems to be using) I'd be especially
> > interested in hearing which features are missing from sbuild that prevent
> > vectis from making use of that functionality.
> 
> The reason vectis doesn't do this is a design choice, not an omission
> in sbuild.

Thanks for your explanation! I now see how vectis is indeed doing something
that is not in the scope of sbuild itself and that vectis indeed has to be a
wrapper around sbuild and other tools.

Though maybe it would be a good idea to add parts of your explanations from
your mail to the README.md and package description. It seems at least me and
Raphael were fooled into thinking that vectis' purpose was a different one than
it actually is.

Thanks for writing this!

cheers, josch


signature.asc
Description: signature

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Simon McVittie]

On Fri, 11 Aug 2017 at 22:26:20 +0200, Raphael Hertzog wrote:
> Simon, you might want to review the history of #859867 and share
> your thoughts about this topic and let us know whether vectis
> is going to handle all this too. :)

Part of the purpose of vectis is to set up and run sbuild in an
environment closely resembling production Debian buildds, including
running sbuild-createchroot, creating the sbuild user with home directory
/nonexistent and so on. Until the production buildds were upgraded to
stretch, it even used buildd.debian.org's special branch/fork of jessie
sbuild (I'm very glad to have been able to get rid of the hacks I used
to make that work).

Unlike what is proposed on #859867, all the setup done by vectis is
inherently disposable. This is a deliberate design choice. The tarball
produced by sbuild-createchroot is cached, but can be regenerated any
time; the autopkgtest virtual machine image likewise (although you have
to be root and use `vectis bootstrap`, vmdebootstrap or some other
manually-created VM to recover if you delete all your cached VM images,
since the same VM image is used to create new VM images in `vectis
new`); and the actual sbuild installation and setup, since it's
sufficiently quick, is done for every build and not cached at all.

This is not necessarily the sbuild that every Debian maintainer wants:
it's somewhat heavyweight (lots of RAM required for virtualization,
lots of setup repeated) and I wouldn't want to build libreoffice
in it. However, it's what *I* wanted, and I only maintain small to
medium-sized packages anyway. vectis is really just automation of things
I was doing already, and things that I would/should have done if they
weren't so much work to do manually (so part of its role is helping me
to live up to the standards the project expects).

The README at https://github.com/smcv/vectis has more details on its
design decisions.

> On Thu, 10 Aug 2017, Raphael Hertzog wrote:
> > We also need
> > qemu images for autopkgtest for example. And they also need to be updated
> > regularly.

`vectis new` creates these images. I haven't yet implemented
`vectis refresh` (rebuild e.g. (VM, sbuild tarball, minbase tarball) x
(sid, buster, stretch, xenial, artful) x (amd64, i386) in a way that
could be done weekly from cron) but I hope to do that soon.

> > And as you mentionned, it would be nice to be able to support derivatives
> > easily, not only in place of Debian, but next to the usual Debian support.

vectis supports Debian and Ubuntu by default, and is reasonably
straightforward to configure for other derivatives (in $dayjob I've
configured it for SteamOS and the Steam Runtime, and one of my design
goals was to make it suitable for other derivatives that Collabora is
involved with, like Apertis).

On Sun, 20 Aug 2017 at 00:26:26 +0900, Johannes Schauer wrote:
> since sbuild already supports autopkgtest as a backend (and thus also
> autopkgtest-virt-qemu which vectis seems to be using) I'd be especially
> interested in hearing which features are missing from sbuild that prevent
> vectis from making use of that functionality.

The reason vectis doesn't do this is a design choice, not an omission
in sbuild.

I have never seriously tried using the autopkgtest backend, because one
of vectis' goals is that if it works in vectis, it should work on the
production Debian infrastructure. Real Debian buildds use the schroot
backend, therefore vectis must use schroot too - because if it didn't,
there would be packages that build successfully in vectis but FTBFS on
production infrastructure, because a typical autopkgtest environment has
fewer strange quirks than a typical sbuild chroot (e.g. the /nonexistent
thing, which has bitten me when working on dbus in the past, and an
atypically small package set because no init, bootloader or kernel is
needed inside a sbuild chroot). The sbuild/schroot invocation is layered
inside a VM, because another vectis design goal is that everything that
runs package code is inside a VM.

Similarly, vectis knows how to run autopkgtest with the qemu backend
(directly) and the lxc and schroot backends (inside a VM), because
ci.debian.net currently uses lxc, so vectis should be able to replicate
that: it matters for packages like flatpak, bubblewrap and debootstrap,
where some tests don't work as desired under lxc and must be skipped or
treated as expected failures. I will admit that I don't usually run
tests under all three backends before upload - if I was a perfect
maintainer, then I would, but it's very slow to run the tests for
something like dbus 8 times (build-time, schroot, lxc, qemu, then
the same again for i386 because I can't upgrade src:dbus on my amd64
host system until I've also built libdbus-1-3:i386).

If the real, production Debian infrastructure ever switches to the
autopkgtest backend, so will vectis.

I would be happy to add *non-default* modes to vectis where it does things
that do not mirror the production infrastructure 

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Michael Stapelberg]

On Thu, May 18, 2017 at 11:39 PM, Michael Stapelberg 
wrote:

> Sorry for the late reply.
>
> On Tue, Apr 11, 2017 at 10:39 AM, Johannes Schauer 
> wrote:
>
>> Hi,
>>
>> Quoting Michael Stapelberg (2017-04-08 11:28:12)
>> > One area where sbuild sorely lacks is configuration, though: pbuilder
>> is very
>> > easy to set up, whereas sbuild requires reading through
>> > https://wiki.debian.org/sbuild, performing a bunch of steps, only to
>> end up
>> > with a setup which works fine for unstable, but seems very clumsy when
>> > building packages for experimental or backports.
>> >
>> > One solution to this issue that I can see is to add a new binary
>> > package to src:sbuild which — possibly after a brief debconf prompt —
>> > performs all the necessary steps to end up with a setup that just
>> > works.
>> >
>> > What are your thoughts on this? Would patches be welcome to add such a
>> > package?
>>
>> patches totally welcome! :D
>>
>
> Cool! I’ll see if I can whip up such a package by working through the wiki
> page.
>

Find attached the first draft of my suggestion. I implemented it as a
separate package purely so that I can build it more quickly, but I assume
we’d want to fold this into src:sbuild eventually.

The resulting package (I built it using dpkg-buildpackage -b) depends on
sbuild, schroot, debootstrap, perl-base, lintian. Upon installation, it
will create an unstable sbuild schroot, modify its configuration, add all
users to sbuild and create a modified ~/.sbuildrc for all users.

If you want to read through the entire behavior, I recommend the
entrypoints debian/postinst and update-sbuild-chroots.

I tested this package on my notebook, which is a Debian installation on
which I never had sbuild installed, so I’m reasonably confident that the
package works — at least to the point that one gets an sbuild installation
that builds packages.

I’d be happy about any feedback. Thanks!


>
>
>>
>> This is a nice idea!
>>
>> Maybe these packages could be named sbuild-backend-${foo} where $foo is
>> the
>> respective backend? At first, a package sbuild-backend-schroot would be
>> cool
>> because schroot is the default backend. It would be nice if that would
>> set up
>> sbuild schroots for stable-backports, unstable and experimental. Maybe
>> that
>> package should also install and activate cron-jobs to regularly update
>> those
>> schroots?
>>
>
> Which other backends even are there, and do we really need to offer our
> users that choice, when we’re talking about a package with sane defaults
> for people who prefer not to make these choices right now? :)
>
>
>>
>> What irks me is, that this setup would be Debian specific. It doesn't
>> make much
>> sense for Debian's downstreams to have have schroots for Debian. Maybe the
>> distribution name should be part of the package name? Or maybe it should
>> be
>> easy for downstreams that care to override the set of distributions the
>> schroots are created for?
>>
>
> Putting Debian into the package name seems reasonable.
>
> How about sbuild-debian-dev-setup?
>
> I originally thought of sbuild-debian-setup, but it could be confused to
> mean “a setup which reflects Debian’s official setup”, i.e. the buildds.
>
> The “setup” suffix to me conveys that this package offers only
> configuration, not new software. Perhaps there’s a more idiomatic term for
> that in Debian?
>
>
>>
>> Thanks!
>>
>> cheers, josch
>>
>
>
>
> --
> Best regards,
> Michael
>



-- 
Best regards,
Michael


sbuild-debian-setup.tar.bz2
Description: BZip2 compressed data

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Michael Stapelberg]

Sorry for the late reply.

On Tue, Apr 11, 2017 at 10:39 AM, Johannes Schauer  wrote:

> Hi,
>
> Quoting Michael Stapelberg (2017-04-08 11:28:12)
> > One area where sbuild sorely lacks is configuration, though: pbuilder is
> very
> > easy to set up, whereas sbuild requires reading through
> > https://wiki.debian.org/sbuild, performing a bunch of steps, only to
> end up
> > with a setup which works fine for unstable, but seems very clumsy when
> > building packages for experimental or backports.
> >
> > One solution to this issue that I can see is to add a new binary
> > package to src:sbuild which — possibly after a brief debconf prompt —
> > performs all the necessary steps to end up with a setup that just
> > works.
> >
> > What are your thoughts on this? Would patches be welcome to add such a
> > package?
>
> patches totally welcome! :D
>

Cool! I’ll see if I can whip up such a package by working through the wiki
page.


>
> This is a nice idea!
>
> Maybe these packages could be named sbuild-backend-${foo} where $foo is the
> respective backend? At first, a package sbuild-backend-schroot would be
> cool
> because schroot is the default backend. It would be nice if that would set
> up
> sbuild schroots for stable-backports, unstable and experimental. Maybe that
> package should also install and activate cron-jobs to regularly update
> those
> schroots?
>

Which other backends even are there, and do we really need to offer our
users that choice, when we’re talking about a package with sane defaults
for people who prefer not to make these choices right now? :)


>
> What irks me is, that this setup would be Debian specific. It doesn't make
> much
> sense for Debian's downstreams to have have schroots for Debian. Maybe the
> distribution name should be part of the package name? Or maybe it should be
> easy for downstreams that care to override the set of distributions the
> schroots are created for?
>

Putting Debian into the package name seems reasonable.

How about sbuild-debian-dev-setup?

I originally thought of sbuild-debian-setup, but it could be confused to
mean “a setup which reflects Debian’s official setup”, i.e. the buildds.

The “setup” suffix to me conveys that this package offers only
configuration, not new software. Perhaps there’s a more idiomatic term for
that in Debian?


>
> Thanks!
>
> cheers, josch
>



-- 
Best regards,
Michael

Bug#859867: [buildd-tools-devel] Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Johannes Schauer]

Hi,

Quoting Michael Stapelberg (2017-04-08 11:28:12)
> One area where sbuild sorely lacks is configuration, though: pbuilder is very
> easy to set up, whereas sbuild requires reading through
> https://wiki.debian.org/sbuild, performing a bunch of steps, only to end up
> with a setup which works fine for unstable, but seems very clumsy when
> building packages for experimental or backports.
> 
> One solution to this issue that I can see is to add a new binary
> package to src:sbuild which — possibly after a brief debconf prompt —
> performs all the necessary steps to end up with a setup that just
> works.
> 
> What are your thoughts on this? Would patches be welcome to add such a
> package?

patches totally welcome! :D

This is a nice idea!

Maybe these packages could be named sbuild-backend-${foo} where $foo is the
respective backend? At first, a package sbuild-backend-schroot would be cool
because schroot is the default backend. It would be nice if that would set up
sbuild schroots for stable-backports, unstable and experimental. Maybe that
package should also install and activate cron-jobs to regularly update those
schroots?

What irks me is, that this setup would be Debian specific. It doesn't make much
sense for Debian's downstreams to have have schroots for Debian. Maybe the
distribution name should be part of the package name? Or maybe it should be
easy for downstreams that care to override the set of distributions the
schroots are created for?

Thanks!

cheers, josch


signature.asc
Description: signature