Re: Announce: docker-buildpackage

[Jeremy Stanley]

On 2018-05-03 23:18:44 +0200 (+0200), Thomas Goirand wrote:
[...]
> Still, this kind of setup (ie: patch proposal and review through a
> review system) was super nice, and I wish we could generalize it by
> plugging Salsa to something like this.
[...]

I agree, it's really nice (I'm probably biased though) and would
love to see Debian be able to take advantage of similar
functionality. I wish my hands weren't already too full to help
maintain something along those lines.

That said, the Zuul maintainers have seen some interest expressed
for adding a Gitlab driver similar to its Gerrit and GitHub drivers,
so could eventually be added as an opt-in solution for Salsa repos
if someone has the bandwidth to run an instance (and sooner if
someone actually volunteers for writing the Gitlab driver!).
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Announce: docker-buildpackage

[Thomas Goirand]

On 05/02/2018 02:02 PM, Ian Jackson wrote:
> And it is for those same reasons that libvirt and openstack are not
> good alternatives.  No-one wants `sbuild-update -udcar unstable' to
> have to involve a libvirt driver for sbuild chroots.

Well, it all depends. If one has to setup an OpenStack deployment to be
able to build packages, then yes, it's too complicated.

But if everyone is using a single access to many public clouds, with a
pool of ready-to-be-used VMs for building, then it's a whole different
thing.

That's what I've been doing to release OpenStack Newton for Stretch. The
experiment was very nice. We unfortunately decided to stop it because
dealing with the way upstream OpenStack is maintained was too demanding.
Still, this kind of setup (ie: patch proposal and review through a
review system) was super nice, and I wish we could generalize it by
plugging Salsa to something like this. Having thousands of pre-built VMs
in a pool for gating the build of packages was kind of cool. Having
packages already built with a temporary repository when someone does a
patch proposal is also super nice. It doesn't mater if it's made with
OpenStack or another virtualization layer. The nice thing was using
something like nodepool and zuul for scheduling jobs, and having it the
default way to submit patches.

Cheers,

Thomas Goirand (zigo)

Re: Announce: docker-buildpackage

[Martin Pitt]

Chow Loong Jin [2018-05-03 12:27 +0800]:
> On Wed, May 02, 2018 at 11:23:56AM +0200, Thomas Goirand wrote:
> > [...]
> > Frankly, I don't see the point in writing this kind of software. Sbuild
> > works super well with the overlay backend, and already has throw-able
> > chroots in tmpfs. Adding docker into this doesn't add any new feature,
> > and in some way, is less flexible than the already existing sbuild.
> 
> Something that comes to mind is network isolation, which sbuild still
> doesn't seem to have proper support[1] for:
> 
> [1] 
> https://wiki.debian.org/sbuild#Disabling_network_access_for_dpkg-buildpackage

Not just network, but also process isolation and reducing privileges. schroot
would be way too "leaky" for a production CI system like ci.debian.net or
autopkgtest.ubuntu.com. The existing backend that compare much better to that
are -lxc and -lxd, and IMHO they are superior to docker. lxc and lxd are "full
OS" containers while docker is optimized for application containers and thus
needs some special treatment (like --privileged, which makes the whole thing
rather unsafe and often causes crashes if you try to start udev in the docker
container) to allow really booting a full OS. Usability wise, they are just as
simple as docker too, as linuxcontainers.org has a lot of pre-built OS images
for all kinds of OSes.

So I agree that there is very little point about adding a docker backend other
than "it's possible" (albeit inferior). As long as someone wants to maintain
it, there is little harm in including it.

Martin


signature.asc
Description: PGP signature

Re: Announce: docker-buildpackage

[Johannes Schauer]

Quoting Chow Loong Jin (2018-05-03 06:27:01)
> On Wed, May 02, 2018 at 11:23:56AM +0200, Thomas Goirand wrote:
> > [...]
> > Frankly, I don't see the point in writing this kind of software. Sbuild
> > works super well with the overlay backend, and already has throw-able
> > chroots in tmpfs. Adding docker into this doesn't add any new feature,
> > and in some way, is less flexible than the already existing sbuild.
> 
> Something that comes to mind is network isolation, which sbuild still
> doesn't seem to have proper support[1] for:
> 
> [1] 
> https://wiki.debian.org/sbuild#Disabling_network_access_for_dpkg-buildpackage

sbuild cannot have or not have support for network isolation. Network isolation
is a feature of the backend and not of sbuild. In this case, the default sbuild
backend (schroot) does not have support for it yet. The bug is even linked in
the wiki section you quote.

If you want network isolation today, just pick one of the other backends that
sbuild supports via autopkgtest (the lxc backend probably supports network
isolation). If you want network isolation with the schroot backend, then you
have to improve schroot and not sbuild.

I also think that, if you want a docker builder today, it would be *much*
easier to just add a docker backend to an existing package building software
like pbuilder or sbuild and thus avoid re-implementing all the "package
building" logic and focus on the docker specific things instead.

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Announce: docker-buildpackage

[Chow Loong Jin]

On Wed, May 02, 2018 at 11:23:56AM +0200, Thomas Goirand wrote:
> [...]
> Frankly, I don't see the point in writing this kind of software. Sbuild
> works super well with the overlay backend, and already has throw-able
> chroots in tmpfs. Adding docker into this doesn't add any new feature,
> and in some way, is less flexible than the already existing sbuild.

Something that comes to mind is network isolation, which sbuild still
doesn't seem to have proper support[1] for:

[1] 
https://wiki.debian.org/sbuild#Disabling_network_access_for_dpkg-buildpackage

-- 
Kind regards,
Loong Jin


signature.asc
Description: PGP signature

Re: Announce: docker-buildpackage

[Thomas Goirand]

On 05/02/2018 02:26 PM, Thadeu Lima de Souza Cascardo wrote:
> On Wed, May 02, 2018 at 11:29:29AM +0200, gregor herrmann wrote:
>> On Wed, 02 May 2018 11:23:56 +0200, Thomas Goirand wrote:
>>
>>> Instead, I very much would prefer a patch to puiparts so that it could
>>> use sbuild's schroot system instead of tarballs.
>>
>> piuparts has support for using chroots:
>>
>> -e dirname, --existing-chroot=dirname
>> Use the specified directory as source for the new chroot,
>> instead of building a new one with debootstrap. This is
>> similar to --basetgz, but the contents are not archived. See
>> also the --hard-link option.
>>
>> I haven't tried it with schroot's chroots yet but it works with
>> cowbuilder's chroots for me.
> 
> And it has --schroot, and just noticed that it also has --docker-image.
> I used the former in the recent past, though not the latter.
> 
>--schroot=SCHROOT-NAME
>Use schroot session named SCHROOT-NAME for the testing 
> environment, instead of building a new one with debootstrap.

Nice, I didn't know. It must be quite new then, as I searched for it
some time ago.

Cheers,

Thomas Goirand (zigo)

Re: Announce: docker-buildpackage

[Holger Levsen]

On Wed, May 02, 2018 at 11:23:56AM +0200, Thomas Goirand wrote:
> Frankly, I don't see the point in writing this kind of software. Sbuild
> works super well with the overlay backend, and already has throw-able
> chroots in tmpfs. Adding docker into this doesn't add any new feature,
> and in some way, is less flexible than the already existing sbuild.

as much as I dislike docker myself, the new feature is being able to use
docker, which for many people (who have a working docker setup) is a
really nice new feature.

(in related spirit I personally dont care about latest piuparts now also
supporting docker, but I happily merged that code as I now some user
will be happy this...)


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: Announce: docker-buildpackage

[Martin Pitt]

Hello Ian, all,

Ian Jackson [2018-05-01 18:59 +0100]:
> > https://salsa.debian.org/ci-team/autopkgtest/merge_requests/3
> 
> Oh, excellent.  It looks like it's awaiting a re-review from Martin.

Reviewed now.

> Martin, will you get to this soon, or would you like me to look at it
> and maybe merge it ?  (You may need to give me some permission in
> salsa for that...)

Oh, does reviewing/commenting require that? Anyway, I was about to add you as a
project member, but 18 hours ago someone already did that.

Martin

Re: Announce: docker-buildpackage

[Thadeu Lima de Souza Cascardo]

On Wed, May 02, 2018 at 11:29:29AM +0200, gregor herrmann wrote:
> On Wed, 02 May 2018 11:23:56 +0200, Thomas Goirand wrote:
> 
> > Instead, I very much would prefer a patch to puiparts so that it could
> > use sbuild's schroot system instead of tarballs.
> 
> piuparts has support for using chroots:
> 
> -e dirname, --existing-chroot=dirname
> Use the specified directory as source for the new chroot,
> instead of building a new one with debootstrap. This is
> similar to --basetgz, but the contents are not archived. See
> also the --hard-link option.
> 
> I haven't tried it with schroot's chroots yet but it works with
> cowbuilder's chroots for me.

And it has --schroot, and just noticed that it also has --docker-image.
I used the former in the recent past, though not the latter.

   --schroot=SCHROOT-NAME
   Use schroot session named SCHROOT-NAME for the testing environment, 
instead of building a new one with debootstrap.

   --docker-image=DOCKER-IMAGE
   Use a container created from the docker image DOCKER-IMAGE for the 
testing environment, instead of building a new one with
   debootstrap. This only supports overlay2 for now and it uses the 
MergedDir layer where piuparts can access, add, edit and
   remove files easily.

Cascardo.

> 
> Cheers,
> gregor
> 
> -- 
>  .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
>  : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
>  `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
>`-   BOFH excuse #213:  Change your language to Finnish. 
> 


signature.asc
Description: PGP signature

Re: Announce: docker-buildpackage

[Ian Jackson]

Simon McVittie writes ("Re: Announce: docker-buildpackage"):
> On Wed, 02 May 2018 at 08:21:41 +0200, Johannes Schauer wrote:
> > Unfortunately, according to Martin [1] it is out of scope for
> > autopkgtest to also add support for making persistent changes to
> > the underlying backend. This in turn means, that an operation
> > like:
> > 
> > $ sbuild-update -udcar unstable
> > 
> > will never work for the autopkgtest backend.

As the original designer and author of the adt virt protocol, I
disagree with Martin.

IMO the right thing to do is to have a new adt-virt-server protocol
verb for "please start working with the `golden image'".  Which might
or might not be supported by any particular virt server, of course,
but should be supported by ones that do some kind of snapshots.

I have disagreed with Martin before on the intended scope and
usefulness of the adt-virt protocols.  Yes, this is in some sense
becoming a "virtualisation manager" but it is not competing with
things like libvirt or openstack because it's much simpler due to not
having to know how to configure or create a virt environment.

And it is for those same reasons that libvirt and openstack are not
good alternatives.  No-one wants `sbuild-update -udcar unstable' to
have to involve a libvirt driver for sbuild chroots.

> This has the same predictability issues as upgrading a system

Of course I agree with this but this is what many of us do and it
should be as smooth as possible.

Ian.

Re: Announce: docker-buildpackage

[gregor herrmann]

On Wed, 02 May 2018 11:23:56 +0200, Thomas Goirand wrote:

> Instead, I very much would prefer a patch to puiparts so that it could
> use sbuild's schroot system instead of tarballs.

piuparts has support for using chroots:

-e dirname, --existing-chroot=dirname
Use the specified directory as source for the new chroot,
instead of building a new one with debootstrap. This is
similar to --basetgz, but the contents are not archived. See
also the --hard-link option.

I haven't tried it with schroot's chroots yet but it works with
cowbuilder's chroots for me.

Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   BOFH excuse #213:  Change your language to Finnish. 

Re: Announce: docker-buildpackage

[Thomas Goirand]

On 05/01/2018 03:23 PM, Enrico Weigelt, metux IT consult wrote:
> Hi folks,
> 
> 
> I've written a tool for isolated deb builds in docker containers.
> It's a little bit like pbuilder, but using docker for isolation.
> 
> https://github.com/metux/docker-buildpackage
> 
> Everything written in shellscript, simple config as sh includes.
> Not debianized yet, as it might require some local customizations.
> (planned for future releases)
> 
> I'm also hacking on another tool which automatically clones repos
> and calls dck-buildpackage for building whole pipelines - but that's
> still experimental and hackish:
> 
> https://github.com/metux/deb-pkg
> 
> 
> --mtx

Hi,

Frankly, I don't see the point in writing this kind of software. Sbuild
works super well with the overlay backend, and already has throw-able
chroots in tmpfs. Adding docker into this doesn't add any new feature,
and in some way, is less flexible than the already existing sbuild.

Instead, I very much would prefer a patch to puiparts so that it could
use sbuild's schroot system instead of tarballs.

Cheers,

Thomas Goirand (zigo)

Re: Announce: docker-buildpackage

[Simon McVittie]

On Wed, 02 May 2018 at 08:21:41 +0200, Johannes Schauer wrote:
> Unfortunately, according to Martin [1] it is out of scope for autopkgtest to
> also add support for making persistent changes to the underlying backend. This
> in turn means, that an operation like:
> 
> $ sbuild-update -udcar unstable
> 
> will never work for the autopkgtest backend.

This has the same predictability issues as upgrading a system rather
than reinstalling it: how do we know that the result of an upgrade is
the same as the result of reinstallation? For real systems we support
upgrades anyway, because the value of carrying over configuration changes
is greater than the cost of some unpredictability; but sbuild chroots
don't/shouldn't have configuration (all the configuration is outside
the chroot on the host system), so that doesn't really apply.

In general an upgrade won't do the same thing as a reinstall, because
old packages that used to be important but are no longer will tend to
remain installed:

- packages that used to be Essential but are not any more,
  because full systems need them but containers and chroots don't
  (init; hopefully sysvinit-utils and e2fsprogs at some point)

- packages that used to be transitively (build-)Essential or important
  but are no longer the preferred choice (old major versions of gcc;
  maybe old major versions of Python if a version of Python becomes
  build-essential)

- superseded versions of libraries (older SONAMEs of libasan, libpcre,
  libncurses, etc.)

As far as I'm aware, the production buildds never upgrade their chroots:
instead they re-bootstrap and discard the old version. Docker strongly
encourages the same approach.

In my build tool Vectis , which uses
autopkgtest virtual machines, I use the same approach for virtual machine
images, sbuild tarballs, piuparts tarballs and lxc images: never upgrade,
only replace. I don't currently use autopkgtest as a sbuild backend,
because one of the design goals of Vectis is that it defaults to the
same mechanisms as the production infrastructure, so that if it works
locally for me it will work on Debian machines too; but it does know
how to re-bootstrap sbuild tarballs separately.

smcv

Re: Announce: docker-buildpackage

[Johannes Schauer]

Quoting Ian Jackson (2018-05-01 16:38:28)
> Geert Stappers writes ("Re: Announce: docker-buildpackage"):
> > On Tue, May 01, 2018 at 09:41:13PM +0800, Paul Wise wrote:
> > > On Tue, May 1, 2018 at 9:23 PM, Enrico Weigelt, metux IT consult wrote:
> > > > I've written a tool for isolated deb builds in docker containers.
> > > > It's a little bit like pbuilder, but using docker for isolation.
> > > >
> > > > https://github.com/metux/docker-buildpackage
> > > 
> > > Does it have any advantages over whalebuilder?
> > 
> > https://tracker.debian.org/pkg/whalebuilder
> > 
> > Homepage: https://www.uhoreg.ca/programming/debian/whalebuilder
> > Debconf talk: https://debconf17.debconf.org/talks/84/
> 
> Do either of these things provide an autopkgtest virt server ?
> 
> That's an executable you can run and speak a protocol to do virty
> kinds of things.
> 
> sbuild and autopkgtest can use any virt system provided in that form.

Unfortunately, according to Martin [1] it is out of scope for autopkgtest to
also add support for making persistent changes to the underlying backend. This
in turn means, that an operation like:

$ sbuild-update -udcar unstable

will never work for the autopkgtest backend.

I still think that it's a better idea to add another autopkgtest backend than
to write yet another sbuild/pbuilder-like tool. But for the autopkgtest backend
to really take off as an sbuild backend, maybe it would be possible to write a
tool that provides a common interface to make persistent changes to autopkgtest
backends? I don't know if it's possible for a third party application to plug
into autopkgtest and offer such functionality as part of its API?

Thanks!

cheers, josch

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886332#20


signature.asc
Description: signature

Re: Announce: docker-buildpackage

[Ian Jackson]

Paul Gevers writes ("Re: Announce: docker-buildpackage"):
> On 01-05-18 16:38, Ian Jackson wrote:
> > Why not write adt-virt-docker ?
> 
> Is that maybe found here:
> https://salsa.debian.org/ci-team/autopkgtest/merge_requests/3

Oh, excellent.  It looks like it's awaiting a re-review from Martin.

Martin, will you get to this soon, or would you like me to look at it
and maybe merge it ?  (You may need to give me some permission in
salsa for that...)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Announce: docker-buildpackage

[Chris Lamb]

[dropping d...@lists.dyne.org, ubuntu-de...@lists.ubuntu.com from CC]

Hi Enrico,

> I've written a tool for isolated deb builds in docker containers.
> It's a little bit like pbuilder, but using docker for isolation.
> 
> https://github.com/metux/docker-buildpackage

Neat. http://travis.debian.net/ uses Docker for its builds - would
I be able to move it to this?


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Announce: docker-buildpackage

[Paul Gevers]

Hi

On 01-05-18 16:38, Ian Jackson wrote:
> Why not write adt-virt-docker ?

Is that maybe found here:
https://salsa.debian.org/ci-team/autopkgtest/merge_requests/3

Paul



signature.asc
Description: OpenPGP digital signature

Re: Announce: docker-buildpackage

[Ian Jackson]

Geert Stappers writes ("Re: Announce: docker-buildpackage"):
> On Tue, May 01, 2018 at 09:41:13PM +0800, Paul Wise wrote:
> > On Tue, May 1, 2018 at 9:23 PM, Enrico Weigelt, metux IT consult wrote:
> > > I've written a tool for isolated deb builds in docker containers.
> > > It's a little bit like pbuilder, but using docker for isolation.
> > >
> > > https://github.com/metux/docker-buildpackage
> > 
> > Does it have any advantages over whalebuilder?
> 
> https://tracker.debian.org/pkg/whalebuilder
> 
> Homepage: https://www.uhoreg.ca/programming/debian/whalebuilder
> Debconf talk: https://debconf17.debconf.org/talks/84/

Do either of these things provide an autopkgtest virt server ?

That's an executable you can run and speak a protocol to do virty
kinds of things.

sbuild and autopkgtest can use any virt system provided in that form.

And other tools which can rely on that interface automatically get
support for:
  schroot lxc lxd qemu
and for running in another environment (without snapshots) via
  chroot ssh
and of course there is also
  null
which just runs things in the calling environment.

/usr/share/doc/autopkgtest/README.virtualisation-server.html in
autopkgtest.deb is the protocol doc - but of course a human user
doesn't have to speak that.  Here's the manpage for adt-virt-schroot:
https://manpages.debian.org/jessie/autopkgtest/adt-virt-schroot.1.en.html

I don't really know why one would want to build a new tool of the form
"does tests or builds or something with testbeds managed by my pet
virtualisaton or container scheme".  Why not write adt-virt-docker ?

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Announce: docker-buildpackage

[Geert Stappers]

On Tue, May 01, 2018 at 09:41:13PM +0800, Paul Wise wrote:
> On Tue, May 1, 2018 at 9:23 PM, Enrico Weigelt, metux IT consult wrote:
> 
> > I've written a tool for isolated deb builds in docker containers.
> > It's a little bit like pbuilder, but using docker for isolation.
> >
> > https://github.com/metux/docker-buildpackage
> 
> Does it have any advantages over whalebuilder?
> 

https://tracker.debian.org/pkg/whalebuilder

Homepage: https://www.uhoreg.ca/programming/debian/whalebuilder
Debconf talk: https://debconf17.debconf.org/talks/84/


Groeten
Geert Stappers
-- 
Leven en laten leven

Re: Announce: docker-buildpackage

[Paul Wise]

On Tue, May 1, 2018 at 9:23 PM, Enrico Weigelt, metux IT consult wrote:

> I've written a tool for isolated deb builds in docker containers.
> It's a little bit like pbuilder, but using docker for isolation.
>
> https://github.com/metux/docker-buildpackage

Does it have any advantages over whalebuilder?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Announce: docker-buildpackage

[Enrico Weigelt, metux IT consult]

Hi folks,


I've written a tool for isolated deb builds in docker containers.
It's a little bit like pbuilder, but using docker for isolation.

https://github.com/metux/docker-buildpackage

Everything written in shellscript, simple config as sh includes.
Not debianized yet, as it might require some local customizations.
(planned for future releases)

I'm also hacking on another tool which automatically clones repos
and calls dck-buildpackage for building whole pipelines - but that's
still experimental and hackish:

https://github.com/metux/deb-pkg


--mtx

--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287