Re: Seeking feedback on a meta package builder
Hi Alex, On Wed, Jun 02, 2021 at 05:07:16PM +0200, Alex Muntada wrote: > This reminded me of my using of ccache and eatmydata in sbuild. > They may not be relevant for the context of mdbp depending on > the resources available for building, indeed. > > I'm pretty fond of them because they speed up a lot some heavy > builds on my laptop, therefore I'd appreciate to have some speed > toggles in any service where I want to send those builds to. TL;DR: Don't add speedup flags, make it fast by default. I don't think speedups are relevant for any of the prospective users. ccache either needs a small set of packages or a huge cache directory to perform well. For many use cases (rebootstrap in particular), the toolchain changes frequently, so the cache would have to be invalidated very often. Most mdbp users would work with at least hundreds of packages. As for eatmydata, I don't think that adding this as toggle is useful. After all, we don't want eatmydata to have any influence on the build. So a mdbp backend can just unconditionally enable it if it makes sense without concerning a prospective user of the API. A very reliably alternative to eatmydata is building on tmpfs with a huge swap partition. It also performs significantly better than eatmydata in my experience as it fully eliminates the need to write to disk a lot of the time. When you use pbuilder's "extract tarball" approach or a type=file schroot with sbuild on a tmpfs (optionally speeding things up by using zstd as compressor), the chroot setup can feel quite instantaneous. Ultimately, speedups should just become the default for the benefit of everyone rather than opt-in. If I remember correctly, pbuilder now defaults to disable man database generation via debconf, which also provides a useful speedup. Helmut
Re: Seeking feedback on a meta package builder
Hi Helmut, > On Mon, May 31, 2021 at 6:26 AM Helmut Grohne wrote: > > > Some aspects that you may want to set for a build are: Paul said: > What to run in the build environment at each stage. This reminded me of my using of ccache and eatmydata in sbuild. They may not be relevant for the context of mdbp depending on the resources available for building, indeed. I'm pretty fond of them because they speed up a lot some heavy builds on my laptop, therefore I'd appreciate to have some speed toggles in any service where I want to send those builds to. Cheers, Alex -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Alex Muntada ⢿⡄⠘⠷⠚⠋ Debian Developer 🍥 log.alexm.org ⠈⠳⣄ signature.asc Description: PGP signature
Re: Seeking feedback on a meta package builder
Hi Paul, On Tue, Jun 01, 2021 at 01:00:55AM +, Paul Wise wrote: > > Some aspects that you may want to set for a build are: > > Looking at my pbuilder configs, some more: Note that you are not a computer program. The thing I'm working on is not supposed to be run by humans, but by other tools. As a human, please continue to use the underlying builder directly to exert the customizations that you need. > Where to place the build results. Covered by mdbp. > Which dependency resolver to use when installing deps. Unsure. This is an aspect where the various build implementations differ. Achieving uniformity on this is hard. > Whether suite/backports to build under. Base suite is required by mdbp. > Which packages not yet in Debian to make available to the build process. I know this isn't the same, but an extra repository gives close to the same flexibility at less convenience. However, convenience is less relevant if the other side is a machine instead of a human. > Extra environment variables to set. Covered by mdbp. > Set the parallelism level. Covered by mdbp. > What to run in the build environment at each stage. Unsure. This also is an aspect where various tools differ widely. Much of the time, you can achieve this by other means. For instance, you can install a special package with a strange postinst. Or you can build a modified source package. Do you have a case where this flexibility really is needed and cannot be achieved by other means? Helmut
Re: Seeking feedback on a meta package builder
Hi Raphaël, On Wed, Jun 02, 2021 at 09:12:24AM +0200, Raphael Hertzog wrote: > On Mon, 31 May 2021, Helmut Grohne wrote: > > I see this tight coupling as a problem for mainly two reasons. > [...] > > * As tasks grow, there is a desire to distribute builds to multiple > >machines. As such, the various tools typically grow their own > >strategy for moving build tasks between machines. > > We already exchanged a while ago about "debusine", it ought to solve this > part a least. > > https://freexian-team.pages.debian.net/debusine/ Yes, we did. > We're just starting to work on the project with the goal of handling > package builds as a first step: > https://salsa.debian.org/freexian-team/debusine/-/milestones/1 > > I invite you to "watch" the project on > https://salsa.debian.org/freexian-team/debusine so that you get > notification of comments on issues as that's where we are discussing > the design. > > And feel free to file feature requests. I think debusine does more and less than what I want at the same time: * debusine thinks of a build more generically than building Debian packages. Building a Debian package just happens to be a special case that is very relevant to debusine, but it shall be able to build other kinds of things as well. * Beyond building, debusine also handles scheduling of builds and storing of artifacts. * debusine's build milestone implements a lock-in on sbuild. The abstraction that I'm seeking doesn't happen here. Possibly debusine could replace its sbuild task with a mdbp task? It would automatically work on pbuilder and mmdebstrap for free. It also doesn't have to implement another method for issuing builds remotely and can concentrate on the scheduling part. > In the current design > (https://salsa.debian.org/freexian-team/debusine/-/issues/7) for the > current iteration, I don't have that level of details but it should > not be too hard to add this on top. Only the differentiated handling of > build/host architecture might be non-trivial as it will impact the scheduling > (identification of an appopriate worker). Please get the architecture part right from the start. It is not that hard actually. Your issue already mentions an unspecific "architecture" field, so you already get to choose suitable workers. It is only the build architecture that affects worker choice. The host architecture is only a setting to pass down and debusine does not have to care about it at all. > > * Does the problem scope I've chosen make sense? > > "Package Build as a Service" :-) Kinda. That would be awesome, yes. Indeed, the abstraction I am thinking of should allow you to swap out a local builder for PBaaS as a simple configuration change. Maybe we can use some AWS credits to provide PBaaS to developers at some point. Or even archive rebuilds as a service. > When thinking of debusine, I thought something of something similar, I > have set out to use sbuild in the first iteration because that what I > (and Debian) uses for official package builds but given the number of > different ways to build packages, I was wondering whether I should > consider to have a tool-agnostic "PackageBuild" task that can be scheduled > and that could then be performed by tool-specific implementations. While Debian uses sbuild in official buildds, reproducible.d.n is fully built on pbuilder instead. It isn't that homogeneous in the end. In order to make debusine easier to set up, I think it should not tightly couple with sbuild. Consider using mdbp to implement your tool-agnostic Debian package building task. What is missing on the mdbp side to make this work for you? Helmut
Re: Seeking feedback on a meta package builder
Hi, On Mon, 31 May 2021, Helmut Grohne wrote: > I see this tight coupling as a problem for mainly two reasons. [...] > * As tasks grow, there is a desire to distribute builds to multiple >machines. As such, the various tools typically grow their own >strategy for moving build tasks between machines. We already exchanged a while ago about "debusine", it ought to solve this part a least. https://freexian-team.pages.debian.net/debusine/ We're just starting to work on the project with the goal of handling package builds as a first step: https://salsa.debian.org/freexian-team/debusine/-/milestones/1 I invite you to "watch" the project on https://salsa.debian.org/freexian-team/debusine so that you get notification of comments on issues as that's where we are discussing the design. And feel free to file feature requests. > So I set out to solve this. More precisely, I looked for an API that > could abstract both sbuild and pbuilder (because they're the most common > ones) and allow executing builds on remote machines. A tool performing > builds should not have to care which one is in use and whether the build > is performed locally or not. Since describing the settings of a build > can be complex, I chose json for representation. Some aspects that you > may want to set for a build are: > * build architecture / host architecture > * arch-only / indep-only / full > * DEB_BUILD_PROFILES > * DEB_BUILD_OPTIONS > * whether network access is permissible > * forcing a build path In the current design (https://salsa.debian.org/freexian-team/debusine/-/issues/7) for the current iteration, I don't have that level of details but it should not be too hard to add this on top. Only the differentiated handling of build/host architecture might be non-trivial as it will impact the scheduling (identification of an appopriate worker). > * Does the problem scope I've chosen make sense? "Package Build as a Service" :-) When thinking of debusine, I thought something of something similar, I have set out to use sbuild in the first iteration because that what I (and Debian) uses for official package builds but given the number of different ways to build packages, I was wondering whether I should consider to have a tool-agnostic "PackageBuild" task that can be scheduled and that could then be performed by tool-specific implementations. Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
